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DON PROCUREMENT METRICS EVALUATION 


ABSTRACT 

The purpose of this MBA Project was to investigate and provide a comprehensive 
evaluation of the current Department of the Navy Procurement Metrics that are collected. These 
metrics are collected by the Office of the Deputy Assistant Secretary of the Navy for Acquisition 
Management. This project was conducted at their request and with their support. The goal of 
this project was to determine if the current metrics are the appropriate procurement performance 
measures. Specifically, this MBA Project attempted to answer these three questions: (1) Do the 
metrics align with strategy? (2) Can they be measured effectively? and (3) Are they linked to 
value? The framework used to explore these questions is Robert Simon’s Levers of Control 
model. These procurement metrics are part of a Diagnostic Control System and are being 
evaluated as such. Simon’s Nature of Measures model is also used in the analysis and helps 
determine metric objectiveness, completeness and responsiveness. 

Keywords: Management Control Systems, Diagnostic Control Systems, Procurement 
Metrics, Performance Measures, Nature of Measures 
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I. 


INTRODUCTION 


Data collection and availability has opened up many opportunities to use that data in a 
way that helps us become better managers. There are several issues surrounding the use of data 
and specifically metrics. For instance, is the data accurate? Is that data in a form that makes it 
reasonable to expend the energy retrieving it? Is the expended energy (resources) worth the 
information that is gleaned from the data? 

The purpose of this project is to provide an evaluation of the current Department of the 
Navy Procurement performance measurements (metrics) being collected from the Major Naval 
Systems Commands (SYSCOMs). Specifically, the metrics are being collected from the Head of 
Contracting Activities (HCAs) within those SYSCOMs. This evaluation will hopefully 
contribute to deciding if the current procurement metrics are useful, appropriate, and of value. 

A. BACKGROUND 

In fiscal year 2002 the SYSCOMs were the subject of a Booz Allen and Hamilton 
effectiveness and efficiency study. The study resulted in eleven recommendations: nine for 
internal spend and two for industry spend. Of these eleven recommendations two applied 
specifically to the procurement community. Within these two recommendations, Booz Allen and 
Hamilton identified an “enabler” for the procurement community. The enabler was “define and 
institute contracting performance measures and reporting” (Procurement Metrics Guide [PMG], 
vl4.3, p. 4). 

The office of the Assistant Secretary of the Navy for Research, Development and 
Acquisition, ASN (RDA), then tasked the office of the Deputy Assistant Secretary of the Navy 
for Acquisition Management, DASN ACQ, to “provide a ‘dashboard’ or spreadsheet for 
examining the effectiveness and efficiency of the Department of the Navy’s Material 
establishment” (PMG, vl4.3, p. 4). DASN ACQ created a working group consisting of HCAs 
representatives from ten SYSCOMs to assist in the development of standard procurement 
performance measures. 
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The working group decided on five focus areas as part of their Balanced Scorecard 1 
approach to developing the procurement performance measures. The five focus areas are 
Customer, People, Process, Financial, and Value. Each area has a strategic theme, objectives, 
and associated metrics. The dashboard was designed for use specifically within the Navy’s 
Procurement Community, both Military and Civilian, and to serve as an executive level summary 
of the Navy’s procurement performance. 

The metrics put together by the working group measure the procurement activities of 
HCA’s representing the following ten SYSCOMs: 


Marine Corps Installations and Logistics (MARCOR I&L) 
Marine Corps Systems Command (MARCORSYSCOM) 
Military Sealift Command (MSC) 

Naval Air Systems Command (NAVAIR) 

Naval Facilities Engineering Command (NAVFAC) 

Naval Sea Systems Command (NAVSEA) 

Naval Supply Systems Command (NAVSUP/FISC’s/ICP’s) 
Office of Naval Research (ONR) 

Space and Naval Warfare Systems Command (SPAWAR) 
Strategic Systems Project Office (SSP) 


B. PROBLEM IDENTIFICATION 

The HCA working group developed 21 metrics that they stated, when used, would give 
an overall picture of the effectiveness and efficiency of the Navy’s procurement function. The 
Office of the DASN ACQ is currently collecting data for the metrics from the SYSCOMs as well 
as pulling data for the metrics from existing databases. However, the office of the DASN ACQ 
is not receiving all of the HCA responsible metrics from each of the ten SYSCOMs. Some 
simply just do not have the information needed to answer DASN’s metrics data call or do not 
have the necessary resources in place to retrieve the data. This creates an issue. The metrics 
need to be evaluated in order to determine if they are useful, appropriate, and of value. 


1 The Balanced Scorecard is a system developed by Robert S. Kaplan and David P. Norton. 
The scorecard takes an organization’s mission and strategy and translates that into goals and 
measures, both financial and non-financial. The basic scorecard uses four areas for 
measurement: financial, customer, internal business, and innovation and learning. The metrics 
working group adapted the basic scorecard to include five focus areas in order to fit their specific 
needs. 
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However, without a relatively complete set of metrics it is difficult to determine whether or not 
the effort should be made to create and generate a full and accurate metric report from each of 
the ten SYSCOMs. 
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II. LITERATURE REVIEW AND FRAMEWORK USED 


A. MANAGEMENT CONTROL SYSTEMS 

Communication is a critical element of the operations of organizations. Information of 
all kinds is communicated through both formal and informal channels. One reason managers of 
organizations are interested in the information communicated is because it sends signals 
describing the health of the organization. Communication channels also provide a means for the 
managers to disseminate information. Senior managers can use those channels for 
communicating a strategic vision, profit goals or receiving information about organizational 
developments or performance. 

All of this information serves as one of the controls in the organization. Controls can be 
put in place through many methods, varying from internal control systems focused on inventory 
tracking to a company’s mission statement. In Robert Simon’s book, Levers of Control, he 
“focuses primarily on the informational aspects of management control systems—the levers 
managers use to transmit and process information within organizations” (Simons, 1995, p. 5). 

He uses the following definition of management control systems: management control systems 
are the formal, information-based routines and procedures managers use to maintain or alter 
patterns in organizational activities (Simons, 1995, p. 5). 

The information-based management control system involves information transmission 
not only from the top management level to lower levels, but also information flow in the 
opposite direction. The information flow from the lower levels to top management allows for the 
monitoring of implemented strategies and other efforts that were communicated down from top 
management. Figure 1 illustrates this point. 
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Figure 1. Information Needs of Top Managers (Simons, 1995, p. 6) 


As stated above, organizational control can take many forms. A well performing 
organization will likely have some sort of control throughout all facets of their organization. 
This may sound oppressive and stifling to the employee’s innovation and motivation. However, 
this does not have to be the case. Control does not necessarily mean micro-managing. 

Robert Simons’ framework has business strategy as its heart. This is how Simons 
describes the framework: 

Business Strategy—how a firm competes and positions itself vis-a-vis its 
competitors—is at the core of the analysis. The second level introduces four key 
constructs that must be analyzed and understood for the successful 
implementation of strategy: core values, risks to be avoided, critical performance 
variables, and strategic uncertainties. Each construct is controlled by a different 
system, or lever, the use of which has different implications. These levers are: 

(Simons, 1995, p. 6) 


1. belief systems, used to inspire and direct the search for new opportunities; 

2. boundary systems, used to set limits on opportunity-seeking behavior; 
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3. diagnostic control systems, used to motivate, monitor, and reward achievement of 
specified goals; and 

4. interactive control systems, used to stimulate organizational learning and the 
emergence of new ideas and strategies (Simons, 1995, p. 7) 

The following figure is the representation of the Levers of Control, with business strategy 
at the core, surrounded by key constructs to be analyzed and understood, and controlled by the 
four systems, or levers. 

BELIEF BOUNDARY 

SYSTEMS SYSTEMS 



INTERACTIVE DIAGNOSTIC 

CONTROL CONTROL 

SYSTEMS SYSTEMS 

Figure 2. Levers of Control Framework (Simons, 1995, p. 7) 


B. BELIEF AND BOUNDARY SYSTEMS 
1. Belief Systems 

Belief systems, as defined by Simons, “are the explicit set of organizational definitions 

that senior managers communicate formally and reinforce systematically to 
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provide basic values, purpose, and direction for the organization” (Simons, 2000, p. 276). The 
belief system provides the means of control over the organization’s foundational principles and 
ideals. 

These principles and ideals are often articulated by an organization’s founders and 
provide the basis for employee action and motivation when rules and regulations alone will not 
sufficiently address the issue or situation that employees face. They address such things as 
responsibilities to customers and other employees. This set of principles is typically called a set 
of core values. Simons defines core values as “the beliefs that define basic principles, purpose, 
and direction” (Simons, 2000, p. 276). The core values often take the form of a mission 
statement or credo. 

2. Boundary Systems 

Belief systems go a long way to inspire and motivate employees to achieve. They give 
them a sense of purpose to their work. This is beneficial, but there is a limit to what employees 
can and should do. The employees need to be motivated, innovative and serve the customer, but 
they should not pursue an avenue which opens up the business to undue risk. An organization 
needs to understand what risks they want to avoid and apply a set of boundaries to employees to 
minimize the likelihood of that risk occurring. This lever of control, boundary systems, plays an 
integral role in business strategy. Without boundaries the belief system is seriously degraded. 
Enthusiasm can lead to undesired outcomes if left unchecked. The boundary system is like the 
brakes on enthusiasm and innovation. Simons uses the example of brakes on a car: 

Cars have brakes for two reasons. First, and most obvious, they allow the driver 
to slow the car down and stop safely. However, cars have brakes for another 
reason. They give the driver the confidence to go very fast. Imagine a high- 
performance racing car on a speedway. The driver can operate at top speeds only 
if he knows that he can rely on excellent brakes to control the car on right turns. 

Like the fastest cars, managers of high-performance businesses need the best 
brakes to control strategic risks that are an inevitable consequence of driving their 
businesses to their maximum potentials (Simons, 2000, p. 275). 

A boundary system basically tells employees what not to do. It lets the employees know 
which actions management deems unacceptable. This may seem like a hindrance on their 
innovation, but in fact, this is what allows them to be innovative. It allows them to be innovative 
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in a way which reduces the likelihood that an organization will be exposed to risk. Another 
approach that a manager could employ would be one telling employees what to do. This method 
ensures that employees do not perform actions that top management feels are risky. This system 
also limits what individuals can actually do. Now that the employees are told what to do, they 
will come to work, follow procedures, and leave. There is minimal opportunity for innovation 
within this system. 

C. INTERACTIVE AND DIAGNOSTIC CONTROL SYSTEMS 

The following two systems, which complete the levers of control model, mean quite 
literally what their titles portray. Managers use these two systems in quite different ways. 
Interactive control systems require a good deal of interaction between top managers and 
employees discussing possible emerging opportunities or threats to their business given the 
changing market environment. The system’s intent is to use the information gained from these 
interactions to modify strategy as the environment changes. Diagnostic control systems are used 
to diagnose issues or discrepancies which vary from the intended strategy or expected output of 
the organization. Diagnostic control systems are used to monitor performance of critical 
performance variables and monitor the implementation of the organization’s intended strategy. 

1. Interactive Control Systems 

Interactive control systems “are the formal information systems that managers use to 
personally involve themselves in the decision activities of subordinates. Simply stated, [they] are 
the hot buttons for senior managers. They provide the information that the boss pays a lot of 
attention to and are used to create an ongoing dialogue with subordinates” (Simons, 2000, p. 
216). 

Interactive control systems are centered on “strategic uncertainties” or the “emerging 
threats and opportunities that could invalidate the assumptions upon which the current business 
strategy is based” (Simons, 2000, p. 215). These uncertainties, by definition, are not known 
beforehand, they emerge over time. The environment is constantly evolving and changing and 
then so too does the organization if it is to survive. Therefore, business strategy, the core of the 
levers of control model, has to be reviewed and examined to determine if the organization’s 
current strategy will put it in a position to be successful. Top managers need to interact with 

their employees to uncover these strategic uncertainties. 
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Examples of strategic uncertainties are rapidly changing technologies which could 
possibly render the organizations’ value proposition worthless and governmental relaxing or 
tightening of regulations. In the first case, this represents a threat to the organization, but 
strategic uncertainties, as stated above, can be avenues for opportunities for the organization as 
well. Both could be true for the second situation, changing government regulations. This could 
help or hurt the organization. 

One way to identify these uncertainties is to create an environment of open dialogue 
throughout the organization which focuses on examining and revising the business strategy given 
the complex and changing environment. Everyone needs to be on board and focus on the 
questions that need to be asked, thereby ensuring a successful future. This open-dialogue 
environment is created through an interactive control system. 

The use of interactive control systems involves thorough evaluation of internal reports 
and efforts and questioning subordinate’s assumptions in those reports in an effort to challenge 
old ways of thinking. This challenge should spark debate and dialogue and ultimately lead to a 
pooling of information, gained throughout the organization, for top managers to pour over and 
help guide the identification of strategic uncertainties. This method of interactive information 
gathering and use of that information is very dissimilar to how a diagnostic control system is 
used. 


2. Diagnostic Control Systems 

This research project focuses on the final lever of control, the diagnostic control system. 

Simons defines a diagnostic control system as the “formal information systems that managers 

use to monitor organizational outcomes and correct deviations from preset standards of 

performance” (Simons, 2000, p. 209). The purpose of a diagnostic control system is to enable 

management by exception. The care and attention used to construct the system and the validity 

of the system determines whether or not managing by exception can be successful. This system 

is only as good as the measurements used to build it, as will be seen later. 

The basic premise of a diagnostic control system is that of variance analysis. This system 

compares actual output or results with those that were expected or set as goals. If the results are 

within the bounds of what was intended to be accomplished then the manager need not search 

any further because things are on track. If, on the other hand, a result is off target, the manager 
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will note the variance and subsequently investigate possible reasons why the deviations occurred. 
The process is referred to as feedback. This is equivalent to a “cybernetic feedback model,” 
shown in Figure 3. In such a model, every measure has a benchmark or standard that represents 
the expected output. The deviation that is the result of comparing the output to the standard 
provides feedback to the input or process parts of the model. The purpose of the feedback is to 
ensure that performance targets are met in the future by adjusting the inputs. 


Standard 


Inputs 



Process 

c 

i 

l 

K 


i 



Outputs 


Feedback 


Figure 3. Cybernetic Feedback Model (Simons, 2000, p. 61) 

A diagnostic control system is applicable in many situations. These situations include 
using tools such as balanced scorecards, budgets, profit plans and standard cost-accounting 
systems in a diagnostic manner. Simons argues that there are two key reasons for a using 
diagnostic control system: “to implement strategy effectively and conserve scarce management 
attention” (Simons, 2000, p. 209). 

Time is scarce for managers. There are thousands of tasks and limited hours in the day. 
The goal of a diagnostic control system is to encapsulate relevant variables into a user friendly 
format that managers can use in a time-efficient manner to give them a snapshot of the health of 
their organization. Simons uses the example of the speedometer in your car. During a long trip 
it would be incredibly tiresome to be constantly monitoring the speedometer and making small 
adjustments to your speed. All of the time and energy would be used doing this task. Instead, 
the car can be put on cruise control which frees the time to focus on other things. This is in 
essence what a diagnostic control system will do. The parameters are set and only when the 
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output falls out of range is there a signal. When that occurs, then focus should be directed to get 
the output back within bounds. 

The particular variables chosen to be measured must be carefully selected. Simons calls 
these variables, critical performance variables. He defines them as “those factors that must be 
achieved or implemented successfully for the intended strategy of the business to succeed” 
(Simons, 2000, p. 209). The diagnostic control system is in place to monitor whether the 
strategy, through these critical performance variables, is being implemented. This system links 
the strategy to the measurable performance variables. The diagnostic control system is crucial to 
communication and effectively implementing strategy in large organizations. 

Risks in Diagnostic Control Systems 

The hands off nature of a diagnostic control system, after extensive attention and work to 
ensure a properly constructed and aligned system, is a very desirable attribute. However, this 
system is not without its risks. Simons identifies three risks associated with using a diagnostic 
control system; “measuring the wrong variables, building slack into targets, and gaming the 
system” (Simons, 2000, pp. 212-213). 

Measuring the Wrong Variables People pay attention to things upon which they 
are evaluated. If senior management identifies incorrect or misaligned variables to be measured, 
then subordinates will be concentrating on achieving goals that do not significantly contribute to 
the overall intended business strategy. Measuring the wrong variables could hinder or derail 
strategy implementation. 

Building Slack into Targets If employees’ performance is evaluated on the 
achievement of a goal, then their natural tendency will be to increase the likelihood that they 
achieve that goal. This can be achieved in one of two ways: (1) through hard work or (2) by 
setting easily attainable goals. Managers need to be cognizant of the latter behavior and ensure 
that the goals that are set are challenging ones. 

Gaming the System In general, employees will work hard towards 
accomplishing goals upon which they are measured. This hard work can be focused away from 
achieving the desired goal in the manner envisioned by top management. “This misdirected 


effort is called gaming” (Simons, 2000, p. 213). Simons (2000) identifies three common gaming 

activities: smoothing, biasing, and illegal acts. The act of “smoothing” attempts to show better 
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performance by manipulating the timing and recording of transactions. An example of this 
might occur if the goals for small business contract awards (in a given fiscal year) are reached. 
Rather than make the current year look better by recording additional small business contract 
awards, after the goal is met, the employee may chose to record any additional contract awards 
towards next year’s goals. “Biasing” occurs when managers chose to report goals that have been 
achieved, but downplay or bury bad news or goals that were not achieved. The attempt is to 
control and bias the information flow. Finally, pressure to achieve can sometimes prove to be 
too much. Individuals may commit “illegal acts” in order to meet their goals and earn a bonus. 
These acts might violate the law or company policy and therefore would be considered illegal. 

D. SELECTING PERFORMANCE MEASURES/METRICS 

In order to mitigate these risks, it is imperative that the correct performance measures or 
metrics are chosen. Performance measures may seem to be intuitive and rudimentary in nature, 
but the opposite is true in many situations. The performance measures selected have profound 
meaning not only for the managers who will be using them to make decisions, but more 
importantly, for those who are being measured. 

Robert Simons identifies three tests which a performance measure must be subjected to in 
order to determine if the measure is a suitable one: (1) Does the measure align with strategy? (2) 
Can it be measured effectively? (3) Is it linked to value? (Simons, 2000, p. 234) 

Test one is whether a measure is aligned with strategy. When a metric or measure is 
created, the managers creating it must know that by doing so they are telling their employees 
what is important. Why have the measure if it is not meaningful? Thus, a measure must be in 
congruence with the strategy of the organization. This not only reinforces the organization’s 
strategy with employees, but it inherently supports the achievement of the strategy. 

Just as important as its alignment with strategy, a metric must be able to be measured 
effectively, Test 2. There is little or no value to having a metric that is supposed to be telling 
managers one thing, but is giving them something different. In reality the metric might either be 
too difficult or complex to measure or simply the wrong metric. In the later case, the correct 
metric can replace the incorrect one, but the former case is a more difficult one. It requires a 
more in-depth analysis of the strategy and a determination if the ability to capture the essence of 
the measure exists. 
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In Robert Simons’ discussion of Test 2, can it be measured effectively, he mentions three 
distinct adjectives against which measures can be evaluated. He states, “Ideally, measures 
should be objective, complete, and responsive.” (Simons, 2000, p. 235) These are referred to as 
the “Nature of Measures.” 

Managers must have solid information on which to base their decisions. Solid 
information comes from having a clear formula for the metric and unambiguous inputs to the 
formula. When the metric is objective it can be independently verified. 2 If the metric is a 
subjective one, then Simons goes on to say in that situation “trust must be high, because the 
subordinate must have confidence that the subjective judgment is fair and will be used 
appropriately.” (Simons, 2000, p. 236) 

Completeness and responsiveness are not as clearly ascertained. Metrics can have 
varying degrees of each. A metric is an attempt to explain an aspect of the organization. How 
well the metric explains that particular aspect of the organization is its degree of completeness. 
When a metric is complete it captures all of the relevant attributes of success. A responsive 
metric is one that responds to the actions of managers. The process of feedback and variance 
analysis used in a diagnostic control system relies partly on an organization’s ability to make 
corrections to inputs to decrease the variance between the output and the performance standard. 

If the metric is not responsive then managers can not influence the output. This may have 
negative impacts on morale because it can create a situation where managers are being evaluated 
on their ability to meet performance targets, but do not have the ability to directly influence the 
results. 

A classic example which illustrates responsiveness and completeness is the “sales calls” 
example. A manager chooses to evaluate employee sales performance by measuring the number 
of sales calls made each week. The metric is responsive because the employee has influence on 
it. While this can be good, it can also create risk. So, if employees know they are being 
evaluated on the number of sales calls per week then they can easily increase the number of sales 
calls without regard for making an actual sale at each stop. Because this metric fails to capture 
the key factor of actual sales it fails the completeness test. 

2 An objective metric is not the only type of measure that is acceptable. Although a 
subjective metric may not be independently verified to the same extent as an objective one, a 
subjective metric can be very valuable in some cases. 


14 



Typically, lower-level jobs which require more routine work can achieve metrics that are 
objective, complete and responsive in nature. The higher up in the managerial chain, the more 
difficult it is to balance all three. The Nature of Measures figure used by Simons (Figure 4) 
provides a template to follow when evaluating objectivity, completeness, and responsiveness of 
performance metrics. 
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Figure 4. Nature of Measures 


Lastly, the metric has to be linked to value. The information contained in the metric 
should be useful for the managers and should aid in decision making. The information provided 
by the metrics should enable managers to be more effective and in the end, run a more effective 
organization. In essence, the metrics should provide valuable information useful in running the 
business and informative in getting from “A to Z.” 

In order to maximize the benefits gained from measuring performance, managers must 
take care in selecting and evaluating specific performance metrics. Specifically, performance 
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metrics should be aligned with strategy, be objective, complete and responsive, and be linked to 
value. 


16 



III. METHODOLOGY 


The author conducted a series of interviews with the Department of the Navy 
Procurement Metrics points of contact provided by DASN ACQ. DASN ACQ provided points 
of contact for the ten SYSCOMs reporting on the metrics. Eventually six-face-to-face interviews 
were performed over the course of three days. The interviews varied from one-on-one, one-on- 
two, and occasionally involved talking to three or more people. The additional individuals were 
identified by the points of contacts as having valuable insights about the metrics. In total, the 
author interviewed thirteen individuals face-to-face. The interviews were conducted at six 
SYSCOM sites and lasted, on average, an hour and a half. There were no telephone interviews 
or follow-up interviews. 

The interviewees had varied familiarity with the performance metrics and job experience. 
This ranged from approximately 2-3 weeks as the DoN Procurement Metrics point of contact to 
being involved in the process throughout the history of the metrics effort. Regardless of this 
factor, every interviewee offered insight into the metrics. Each interviewee had his or her own 
specific tone about the metrics; however commonality of issues with the metrics was apparent 
across all of the interviews. 

The general framework used to evaluate the metrics consists of three interview questions: 
(1) Do the metrics align with strategy? (2) Can the metrics be measured effectively? and (3) Are 
the metrics linked to value? The interviews provided key information that was useful in 
answering all three of these questions. The information also provided general views on how the 
overall metrics process could be improved and specifically how the metrics could become more 
valuable for the HCAs. 

Table 1 is a summary of the DoN Procurement Metrics evaluated in this project. The 
following section summarizes the interviewees’ and author’s evaluation of the current 
performance metrics. The author’s evaluation of the metrics represents the synergy of 
performance measurement theory and information gained during the interviews. 
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FOCUS 

AREA 

TIER 1 METRIC 

TIER 2 METRIC 

TIER 3 
METRIC 

TIER 4 
METRIC 

DESCRIPTION / FORMULA 

FREQUENCY 

COLLECTED 

BY 

Customer 

Customer Satisfaction Index 




Average Index from Customer Satisfaction Survey 

Annual 

HCA 

People 

Human Capital Index 

Employee Survey 



Average Index from Employee Satifaction Survey 

Annual 

HCA 

Workforce Stability 



Percent difference between authorized end-strength 
and actual employees onboard 

Quarterly 

HCA 

Qualifications 

Continous 

Learning 


Percent employees meet their CL requirements and 
hold valid certificates 

Annual 

DASN ACQ 

DAWIA 


Percent DAWIA certified to or above level required by 
their position 

Quarterly 

DASN ACQ 

Acquisition 

Professional 

Community 


Percent of incumbents of CAPs who are APC 
members 

Quarterly 

DASN ACQ 

Process 

Process Improvement Index 

E2E Metrics 



Percent of automated procurement transactions 
(defined on PEO-ARBS website) 

Quarterly 

DASN ACQ 

Efficiency 

Improvement 

P-Card 

Delinquency 


60 days delinquent/total transactions 

Quarterly 

DASN ACQ 

DD1716 


Average turnaround time from receipt to disposition of 
DD1716 (notice of contract deficiencies) 

Quarterly 

DASN ACQ 

Interest 

Penalties 


Ratio YTD interest penalties/penalties same period 
previous FY 

Quarterly 

DASN ACQ 

Cycle Time 



Average cycle time as the term is defined in glossary 
for ACAT 1 and II Program contracts 

Annual 

HCA 

Consolidate Services 
Contract 



Services contracts awarded YTD/awarded previous 

FY 

Quarterly 

DASN ACQ 

Financial 

Contribution Index 

Procurement 
Direct/Indirect Ratio 



Procurement direct labor dollars/total procurement 
labor dollars 

Annual 

HCA 

Industry Spend 
Analysis 

Competition 


Percent dollars awarded competitively/total available 

Quarterly 

DASN ACQ 

Small Business 


Percent dollars awarded to small businesses/total 
dollars awarded and available for small business 

Quarterly 

DASN ACQ 

Commercial 

Items 

Actions 

Percent contracts awarded for commercial items/total 
contracts awarded 

Quarterly 

DASN ACQ 

Dollars 

Percent dollars awarded for commercial items/ dollars 
awarded for commercial items in '99 

Quarterly 

DASN ACQ 

Performance 
Based Services 
Contracting 

Actions 

Percent actions awarded for performance based 
services/total actions awarded for services 

Quarterly 

DASN ACQ 

Dollars 

Percent dollars awarded for performance based 
services/total dollars awarded for services 

Quarterly 

DASN ACQ 

Value 

Product Unit Costing (PUC) 

Large Contracts 



Procurement salary dollars/contract action 
dollars*(customer service survey index) 2 , for contracts 
> $100k 

Annual 

HCA 

Simplified Acquisition 
Procedures 



Procurement salary dollars/contract action 
dollars*(customer service survey index) 2 , for contracts 
< $100k 

Annual 

HCA 


Table 1. DoN Procurement Metrics Summary 
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IV. PROCUREMENT METRICS ANALYSIS 


The Procurement Metrics analysis is divided into four sections. The first three sections 
answer the three questions described as the general framework for evaluating the metrics: 1) Do 
the metrics align with strategy? 2) Can the metrics be measured effectively? and 3) Are the 
metrics linked to value? The fourth section of this chapter describes other findings that do not 
specifically address the three questions. 


A. DO THE METRICS ALIGN WITH STRATEGY? 


The metrics are categorized into five focus areas; Customer, People, Process, Financial, 
and Value. Each focus area has an associated strategic theme. The first question evaluates 
whether the metrics link back to the overall strategy of each focus area. If they are not aligned 
with strategy then there is a fundamental disconnect. 

1. Customer 


Metric 


Customer Satisfaction Index 


The Customer focus area uses only one metric, the Customer Satisfaction Index. The 
strategic theme is to “Deliver unprecedented offensive power, defensive assurance and 
operational independence to the Joint Force Commanders” (PMG, vl4.3, p. 7). Put another way, 
the strategy is to “give the customers what they want.” The index is an average of the scores 
from customer satisfaction surveys which address four specific customer satisfaction themes 
identified in the Procurement Metrics Guide. The Customer Satisfaction Index metric is aligned 
with strategy; the SYSCOMs want to know how well they are serving their customers. 

However, this does not mean that the metric is flawless. The metric is still subject to the tests of 
whether it is linked to value and can be accurately measured. 
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2. People 


Metrics 

Employee Survey 


Workforce Stability 


Continuous Learning 


DAWIA Certification 


Acquisition Professional Community 


The People focus area has five associated metrics. The five metrics are the Employee 
Surx’ey, Workforce Stability, Continuous Learning, DAWIA Certification, and Acquisition 
Professional Community. The strategic theme is to “Maintain world-class workforce” (PMG, 
vl4.3, p. 7). All of these metrics align with strategy. The employee survey is intended to 
measure the employee’s overall satisfaction with the organization which is an important 
component of maintaining a world-class workforce. A satisfied workforce is likely to be more 
stable and keeping the workforce stable (providing they are well trained) is certainly a part of 
maintaining a world-class workforce. The last three metrics are in place to ensure that 
employees have the knowledge and necessary skills to be effective in their positions, making 
Workforce Stability that much more meaningful. Continuous Learning and the Defense 
Acquisition Workforce Improvement Act (DAWIA) metrics measure the percentage of employees 
who have their required certification. Continuous Learning certification requires job relevant 
“learning” to be achieved each year. The level of DAWIA certification needed depends on the 
position that the employee holds. In order to be certified, employees need to be certified to their 
level position or higher. The Acquisition Professional Community metric, collected quarterly, 
identifies the percentage of incumbents in Critical Acquisition Positions who are members of the 
Acquisition Professional Community (APC) compared to the total number of incumbents in 
Critical Acquisition Positions. The PMG states, “Selection to the APC is based on meeting 
specific training, education, experience and grade requirements.” These requirements are 
controlled by the Director of Acquisition Career Management (DACM). 
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3. Process 


Metrics 

E2E Metric 


P-Card Delinquency 


DD1716 


Interest Penalties 


Cycle Time 


Consolidate Service Contract 


The third focus area is Process. There are six associated metrics in this focus area. The 
strategic theme for Process is to “Continuously improve efficiency” (PMG, vl4.3, p. 7). The 
metrics associated with the Process focus area are E2E Metrics, P-Card Delinquency, DD1716, 
Interest Penalties, Cycle Time, and Consolidate Service Contract. 

Business transactions are moving in the direction of paperless transactions. The Navy 
has set up a website (www.peoarbs.navy.mil) which it refers to as the Paperless Acquisition 
Office of the Navy. The goal of the website is “to simplify and modernize the Navy acquisition 
process in the area of contract writing, administration, finance and auditing.” The E2E Metric 
measures the percentage of total procurement transactions that are automated, or the ratio of 
automated procurement transactions over total number of procurement transactions. Automation 
in this case improves efficiency because it reduces time spent filling out and sending paper 
procurement transactions. This metric links to the strategy of the Process focus area. 

The DoD Purchase Card Program Management Office sent out a memorandum on 17 
SEP 1999 regarding the issue of Purchase Card Delinquency policies and the need to keep 
delinquent payments (over 60 days) to a minimum. The P-Card Delinquency metric is the 
percentage of dollars of delinquent P-card payments compared to the total purchase card dollars. 
The metric is linked with the overall strategy for the Process focus area because decreasing 
delinquent P-Card payments results in an increased efficiency of payments. This metric is 
consistent with DoD policy as well. 

The DD1716 Metric measures the average turnaround time of the DD1716 form, which is 
a contract deficiency report. The quick resolution of the deficiency can limit the amount of 
interest charged. The sooner the turnaround time on these forms the less money that has to be 
paid out and more efficient the process which directly links to the strategy. 
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Interest penalties cost the Navy and the DoD as a whole millions of dollar every year. 
Paying interest is essentially wasted money that results from inefficient practices and/or contract 
mistakes. Not only does the Interest Penalty metric provide the opportunity to monitor dollar 
savings, but it is also a means to monitor if the process is becoming more efficient. Increasing 
dollar savings and efficiency work hand in hand towards the ultimate objective: an efficient 
environment which becomes an effective cost savings tool. This metric is collected quarterly 
and compares the current year’s interest penalties paid to date to the same period of the former 
fiscal year. The output is a ratio which hopefully falls below 100 percent. This metric, as well 
as Consolidate Service Contract and the Industry Spend Analysis—Commercial Items (actions & 
dollars) metrics compare current year to last year, so the base needs to be adjusted every year. 

Cycle time is very important in the procurement world and there are many factors that 
influence it (e.g., technological maturity of the proposed system). Reducing cycle time can save 
money. More importantly, a reduction in cycle time gets the products or services to the war 
fighter sooner. A reduction in cycle time can give the United States military a distinct advantage 
over other militaries. The Cycle Time metric is defined as “total time in days between the 
acquisition start date and end date (only applicable to ACAT I and II acquisitions)” (PMG, 
vl4.3, p. 17). Definitions of the start and end date are available in the Procurement Metrics 
Guide. The Cycle Time metric is linked to the Process focus area strategy. 

The Consolidate Service Contract metric is the last one under the Process focus area. 

This metric is collected quarterly and is measured in a manner that is similar to the Interest 
Penalty metric. It is a ratio of the number of services contracts awarded year-to-date as 
compared to the same time the previous year. The goal for the ratio is for it to be less than 100 
percent, indicating that some of the services contracts have been consolidated or eliminated. 

This metric is aligned with the overall strategy of the Process focus area because fewer contracts 
will have to be maintained and administered and this increases the potential for more efficient 
operations. 
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4. Financial 


Metrics 

Procurement Direct/Indirect Ratio 


Industry Spend Analysis—Competition 


Industry Spend Analysis—Small Business 


Industry Spend Analysis—Commercial Items—Actions 


Industry Spend Analysis—Commercial Items—Dollars 


Industry Spend Analysis—Performance Based Services—Actions 


Industry Spend Analysis—Performance Based Services—Dollars 


The Financial strategic theme is to “Maximize return on taxpayer investment” (PMG, 
vl4.3, p. 7). The financial focus area has seven associated metrics. The metrics are (1) 
Procurement Direct/Indirect Ratio , (2) Industry Spend Analysis — Competition , (3) Industry 
Spend Analysis—Small Business, (4) Industry Spend Analysis—Commercial Items-Actions, (5) 
Industry Spend Analysis—Commercial Items-Dollars, (6) Industry Spend Analysis—Performance 
Based Services Acquisition Contracting-Actions, and (7) Industry Spend Analysis—Performance 
Based Services Acquisition Contracting-Dollars). 

The Procurement Direct/Indirect Ratio metric is an attempt at measuring the “bang for 
the buck” of each SYSCOM when it comes to how much support is needed to perform the 
procurement function. The metric is a ratio of the total number of direct labor employees as 
compared to the total number of employees in the procurement organization. Direct labor 
employees are defined as “employees whose primary responsibilities are directly involved with 
efforts required to perform the procurement function” (PMG, vl4.3, p. 17). The idea is to 
provide the necessary support to efficiently and effectively accomplish all of the necessary 
contracting activities while keeping the number of supporting positions under control. The 
strategy of maximizing the return on taxpayer investment follows logically, but issues with this 
metric are discussed in later sections. 

The Industry Spend Analysis—Competition metric has a direct link to the strategy of the 
Financial focus area, maximize return on taxpayer investment. This metric is the ratio of total 
procurement dollars awarded competitively as compared to the total procurement dollars 
available for competition. The benefits of competition; cost reduction, improved quality, and 
innovation, are widely known. There are instances of course when competition is not the most 
advisable way to proceed (e.g., when only one source is able to provide the product or service, 
when time is a critical factor, or when a dependable long-term relationship is necessary). 
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The Industry Spend Analysis—Small Business metric links to the strategy of the Federal 
Acquisition Regulations regarding small business opportunities for government contracts. This 
metric does not clearly link to the financial focus area strategy of maximizing taxpayer 
investment. However, it does reflect social responsibility and the entrepreneurial spirit captured 
in the Small Business Act. The spirit of the Small Business Act is to encourage small business 
participation in government contracts and specifically, “small business concerns shall be 
afforded an equitable opportunity to compete for all contracts that they can perform to the extent 
consistent with the Government’s interest” (Federal Acquisition Regulation [FAR], 19.202-1). 

The Federal Acquisition Regulation also states: 

“It is the policy of the Government to provide maximum practicable opportunities 
in its acquisitions to small business, veteran-owned small business, service- 
disabled veteran-owned small business, HUBZone small business, small 
disadvantaged business, and women-owned small business concerns. The Small 
Business Administration (SBA) counsels and assists small business concerns and 
assists contracting personnel to ensure that a fair proportion of contracts for 
supplies and services is placed with small business” (FAR, 19.201). 

The Industry Spend Analysis—Commercial Items metrics (Actions & Dollars) both are 
linked to the Financial focus area strategy as well as that of the Federal Acquisition Regulations. 
Some of the benefits of using Commercial and Non-Developmental items (CANDI) is the speed 
at which these items can be procured, cost savings, and proven technology which mitigates risk. 
Using CANDI for technologically driven products maximizes taxpayer investment because often 
times commercial demand for these items may quicken the evolution and fielding beyond the 
pace at which government research and development could produce the same items. The Actions 
metric is the “ratio of the number of contracts awarded year to date for commercial items as 
compared to the total number of contracts awarded” (PMG vl4.3, p. 13). The Dollars metric is 
the “ratio of HCA total product dollars awarded year to date for commercial items as compared 
to the total product dollars awarded in 1999, during the comparable period, for commercial 
items” (PMG, vl4.3, p. 13). 

The Federal Acquisition Regulation states: 

“Agencies shall—(a) Conduct market research to determine whether commercial 
items or nondevelopmental items are available that could meet the agency’s 
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requirements; (b) Acquire commercial items or nondevelopmental items when 
they are available to meet the needs of the agency; and (c) Require prime 
contractors and subcontractors at all tiers to incorporate, to the maximum extent 
practicable, commercial items or nondevelopmental items as components of items 
supplied to the agency” (FAR, 12.101). 

The Industry Spend Analysis—Performance Based Services Acquisition (PBSA) 
Contracting metrics (Actions and Dollars) are aligned with focus area strategy as well as DoD 
policy. The motivation for these metrics is that people working within contracting do not need to 
get bogged down in the details of writing specific processes of how a service should be 
performed. The contractor is supposed to be the expert, so give the contractor specifics on what 
the end result of the service should be, but not the minutia of how to do it. The effects of PBSA 
should be cost savings and a more effective and efficient use of limited DoD resources because 
the focus shifts to execution. 

“PBSA involves acquisition strategies, methods, and techniques that describe and 
communicate measurable outcomes rather than direct performance processes. It 
is structured around defining a service requirement in terms of performance 
objectives and providing contractors the latitude to determine how to meet those 
objectives. Simply put, it is a method for acquiring what is required and placing 
the responsibility for how it is accomplished on the contractor” (Guidebook for 
PBSA in the Department of Defense, p. 6). 

The PBSA Actions metric is defined as the ratio of HCA total PBSA awarded actions as 
compared to total awarded actions for services. The PBSA Dollars metric is defined as the ratio 
of HCA total dollars awarded for PBSA contracts as compared to HCA total dollars awarded for 
all services contracts. 

5. Value 


Metrics 

Performance Unit Costing—Large Contracts 


Performance Unit Costing—Simplified Acquisition Procedures 


The fifth and final focus area is Value. The strategic theme of the Value focus area is the 
same as the Financial focus area: Maximize return on taxpayer investment. The objective for 
this focus area is to “Balance quality, cost and productivity” (PMG, vl4.3, p. 8). There are two 
metrics in this focus area and both support the strategic theme. The metrics are Performance 
Unit Costing (PUC)—Large Contracts and PUC—Simplified Acquisition Procedures. Both 

25 




metrics use the same formula: HCA total procurement salary dollars (C), divided by the product 
of contract action dollars (O) and the square of the customer satisfaction index (I - ). The resultant 
formula is: [C/(0*I)]. The two metrics have different definitions for “O,” the contract action 
dollars, because of the threshold between Large Contracts and Simplified Acquisition 
Procedures. The Simplified Acquisition Threshold is defined by the FAR as follows: 

“Simplified acquisition threshold means $100,000, except for acquisitions of 
supplies or services that, as determined by the head of the agency, are to be used 
to support a contingency operation or to facilitate defense against or recovery 
from nuclear, biological, chemical, or radiological attack” (FAR, 2.101). 

The PUC-Large Contracts metric is measured by taking “HCA total procurement salary 
dollars (C) divided by contract action dollars (O) for contracts valued at greater than $100k 
multiplied by the square of the customer satisfaction index (I): [C/(0*I 2 )]” (PMG vl4.3, 14). 

The PUC-Simplified Acquisition Procedures metric is calculated the same way, just using the 
“$100k and less” threshold to determine the “O” variable. The motivation for these metrics is to 
mesh the “bang for the buck” (cost and productivity) with the perceived quality by the 
customers. The output is a number (e.g. .004959716), which by itself, is of no use. It can be of 
some use if several other factors are taken into account such as the validity of the Customer 
Satisfaction Index, the ability to accurately collect total procurement salary dollars and having 
several data points (a trend) to analyze. 

B. CAN THE METRICS BE MEASURED EFFECTIVELY? 

All of the DoN metrics appear to be aligned with the strategies of the focus areas. One of 
the strategies is to give the war fighters what they need when they need it and a survey could 
possibly capture all that is needed to ascertain that, but it could also fall short. In this section, the 
metrics will be evaluated on their objectivity and completeness as described in the literature 
review. A measure is objective when it can be independently verified and is complete when it 
captures all aspects of construct. Determining the responsiveness of the metrics is beyond what 
can be accomplished in this study. Capturing the responsiveness dimension will require a more 
in depth understanding of the HCA organization structure as well the intricacies and 
responsibilities of the different positions within that organization. To reiterate from the literature 
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review, a responsive measure reflects actions that a manager can directly influence. Most of 
these metrics encompass actions resulting from the work of an entire enterprise and therefore it is 
difficult to ascertain at what level they are responsive or not responsive. 


1. Customer 


Metric 


Customer Satisfaction Index 


Can Customer Satisfaction be measured effectively? It depends on several things. For 
instance: Who is filling out the survey? Is it the same person as last year? How does this person 
typically fill out a survey: accurately, harshly, or lenient? Is this the same survey that was given 
out last year? If not, how similar is it to last years, does it cover the same general customer 
satisfaction themes? Does a survey even go out every year or was last year’s survey responses 
used for this year’s metric? What was your response rate? All of these questions offer a look 
into whether or not the Customer Satisfaction Index metric can be an effective measure as well as 
a tool to analyze trends from the metric. 


All of the aforementioned questions about the surveys identify possible risks in using the 
surveys in trend analysis. Is a survey better than none? Absolutely. Any one of the factors 
identified in the questions by itself is not likely to affect drastically the validity of the Customer 
Satisfaction Index, but put together, they can throw off the baseline and trend analysis. 


The questions mainly address the issue of objectiveness. Surveys are inherently 
subjective, but if they are performed in good faith, address the four customer satisfaction themes, 
are reported on accurately, and have a high response rates, then they should be an effective 
measure of customer satisfaction. 


Is the Customer Satisfaction Index complete; does it capture all the relevant attributes of 
achievement? If the survey covers fully the four customer satisfaction themes required, as well 
as goes to and is received back from a large number of customers (high response rate), as 
described above, then the survey should capture all the attributes of Customer Satisfaction. The 
interviewees stated that customer response rate was not a problem. However, there is currently 
no required response rate delineated in the PMG setting the bar for acceptable survey results. 
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2. People 


Metrics 

Employee Survey 


Workforce Stability 


Continuous Learning 


DAWIA Certification 


Acquisition Professional Community 


All of the questions that can be asked about the Customer Satisfaction Survey can be 
asked about the Employee Survey and they lead in the same direction. The characteristics of an 
objective and complete employee survey can be gleaned from the above discussion of the 
customer satisfaction survey. That is, surveys are inherently subjective, but that characteristic 
can be minimized through the appropriate methods; for instance, requiring a high response rate 
and accurate reporting. No specific percentage for response rate was discussed in the interviews, 
but one interviewee stated, “Need to put in place a certain percentage that has to respond for it to 
be valid. The survey is anonymous, so it is hard to control.” Also, the survey is complete as 
long as it measures all necessary attributes of employee satisfaction. One interviewee mentioned 
that their employee survey is sent to the entire SYSCOM, not just the HCA, so it is not just 
measuring the HCA’s employee satisfaction. 

Workforce Stability is an objective measure unless estimates are used; however, this 
metric is not complete as it currently stands. The metric does not capture significant elements of 
workforce stability; rather it captures just a limited scope. It should be possible to effectively 
measure Workforce Stability , but that is not what is being measured by using this metric. This 
metric measures the percent difference between the authorized end-strength and the actual 
number of employees onboard as of the end date of the quarter. This measures how “full” the 
organization is, not how stable the workforce is. Even with several data points of this metric 
creating a trend, it still is not very useful unless you are assessing how full an organization is 
over time. The measure tangentially relates to stability. If during a period of time the percent 
difference stayed relatively stable, it could be that there was stability in the workforce. It could 
also mean that each time you measured stability using this formula, the number of employees 
and end-strength remained about the same, but that there was a 100 percent turnover of the 
workforce. This is not a stable workforce, it has high turnover which is almost certainly 
detrimental to the organization’s effectiveness. 
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The next three metrics under the People Focus Area, Qualifications: (1) Continuous 
Learning, (2) DAWIA certification, and (3) Acquisition Professional Community can be 
measured effectively. The data for these metrics are pulled by DASN ACQ from the Data 
Acquisition and Control Manager Mission Information System database (DACM MIS database). 
As long as the information is correctly inputted in the database the resulting metric should be 
accurate and therefore make the metric an objective one. These metrics also present a full 
picture of the level of qualifications that the procurement workforce making the measures 
relatively complete. 


3. Process 


Metrics 

E2E Metric 


P-Card Delinquency 


DD1716 


Interest Penalties 


Cycle Time 


Consolidate Service Contract 


The E2E metric can be effectively measured. This metric is objective because 
information for the metric is taken from the same database for each SYSCOM and therefore, can 
be independently verified. As is the case with most of the metrics that are being collected from 
databases, the accuracy of the information entered into the database is paramount to getting an 
accurate reflection of the SYSCOM. The P-Card Delinquency metric is affected by the same 
circumstances as the E2E metric. It is effectively measured through the Navy’s e-Business 
Office and is objective. The Consolidate Service Contract metric is collected by DASN ACQ 
from the Procurement Management Reporting System (PMRS). All three metrics; E2E , P-Card 
Delinquency , and Consolidate Sendee Contract, are narrowly focused and inclusive, making the 
measurement of the metric complete. 

The Efficiency Improvement metric of DD1716 is collected by DASN ACQ from the 
Navy-Air Force Interface (NAFI) website. This metric is relatively objective because the data is 
pulled from the NAFI website, but there is a tinge of subjectivity surrounding what the author 
views as the potential for gaming the system, which is discussed in a later section. This metric, 
if the receiving date entered into the NAFI website is agreed upon, is complete in its measure of 
the efficiency of DD1716 resolution. 
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The issues of objectivity and completeness for the Cycle Time metric are similar to the 
DD1716 metric. This metric is reported by the HCAs instead of being collected from a database, 
increasing its subjectivity. Since there are several determinates when deciding the actual start 
and end date of the metric, the chances of gaming increase. It is a relatively complete measure of 
the start to the end of acquisition time. 


4. Financial 


Metrics 

Procurement Direct/Indirect Ratio 


Industry Spend Analysis—Competition 


Industry Spend Analysis—Small Business 


Industry Spend Analysis—Commercial Items—Actions 


Industry Spend Analysis—Commercial Items—Dollars 


Industry Spend Analysis—Performance Based Services—Actions 


Industry Spend Analysis—Performance Based Services—Dollars 


The Procurement Direct/Indirect Ratio metric is a measure that has potential. As stated 
above in the “Aligning with Strategy” section, this metric is designed to measure how much 
support is needed to carry out the procurement function for each SYSCOM. The metric is likely 
to differ between the SYSCOMs. That is appropriate. One SYSCOM could mainly be making 
small purchases and the other purchasing Aircraft Carriers. The two SYSCOMs in this example 
would have different ratios given a need for different levels of various activities and processes. 
Even if the ratios differ across SYSCOMs, a trend analysis within the SYSCOMs is beneficial. 

Given how this metric is collected, it is not totally objective. There needs to be very 
specific instruction on which positions should be reflected in this metric. Precision and accuracy 
in collecting the data for this metric will ensure it is objective. One interviewee stated, 
“Direct/Indirect [is] a little bit subjective. The information is not easy to get.” Another 
interviewee stated, “Does anyone track procurement salary? Information is there, but not broken 
down into sections like procurement and further into direct and nondirect.” Reflecting the need 
for clarification on definitions in regards to this metric, one interviewee stated, “There is 
confusing language in the definition of terms, people need to clearly understand what is being 
asked for.” This metric gives a relatively complete picture of how much indirect support is 
needed by SYSCOM in order to perform their level of procurement. 

The Industry Spend Analysis—Competition metric can be effectively measured. All of 
the information needed to calculate the metric is in the Procurement Management Reporting 


30 




System (PMRS) and is pulled by DASN ACQ which makes it an objective measure. The same 
also holds true for the following five Industry Spend Analysis metrics: (1) Small Business, (2) 
Commercial Items—Actions, (3) Commercial Items — Dollars, (4) Performance Based Sendees 
Acquisition Contracting—Actions and (5) Performance Based Services Acquisition 
Contracting — Dollars. These metrics are all collected from PMRS by DASN ACQ. All of these 
metrics are complete; they capture all the relevant information that is needed to get a useful view 
of how the Navy’s procurement function is performing in each of these areas. 


5. Value 


Metrics 

Performance Unit Costing—Large Contracts 


Performance Unit Costing—Simplified Acquisition Procedures 


The Performance Unit Costing (PUC) metrics. Large Contracts and Simplified 
Acquisition Procedures, can be measured, but the degree of the measure’s precision is in 
question. First, the Customer Satisfaction Index is used in the formula. This is not a completely 
objective variable, so it adds some subjectivity into these metrics. This index is squared in the 
formula to increase its affect on the output of the equation. 

Second, because the data available for procurement salary dollars to the SYSCOMs 
differs, the use of reasonable estimation is an option given by the PMG. This makes the 
likelihood of independent verification questionable. One interviewee stated, “Procurement 
salaries can be subjective, [you] need an unbiased source that they can pull from.” 

The contract action dollars, variable “O,” is defined as “The total net value of obligations 
under contract actions” (PMG, vl4.3, p. 16). This is a relatively objective measure with no 
estimation needed. 

The PUC metrics are intended to be a complete measure of value and specifically balance 
quality, cost and productivity. The metrics, although somewhat abstract and not very meaningful 
without trend analysis, do blend all three of these elements together using the formula. The HCA 
representatives generally understood the concept and why the formula was created the way it 
was, but were concerned with the meaning of the output. 
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C. ARE THE METRICS LINKED TO VALUE? 

All of the HCAs interviewed thought that feedback from DASN ACQ was significantly 
lacking on (1) the DoN Procurement Metrics in general and (2) specific feedback on the metrics 
DASN ACQ collects on each of the SYSCOMs. In the HCAs view, if they are not collecting the 
data for a metric, then it does not even exist because they have not seen any feedback on the 
fourteen metrics that DASN ACQ collects. Therefore those fourteen metrics are of little value to 
the HCAs, regardless if they have value for DASN ACQ. 

Being the POCs for the DoN Procurement Metrics and collecting the metrics is a 
collateral duty for all of the POCs. They receive no additional resources to aid in the data calls. 
The collection of these metrics is a time-consuming process. Collecting the data for the metrics 
expends valuable resources and it is not clear to the HCAs if the “cost” is worth the benefit or 
value the metrics provide to the SYSCOMs and DASN ACQ. 

An issue brought up in one of the interviews was whether or not DASN ACQ had any 
idea of how much it cost to develop, monitor and collect the data on the metrics. So, what is the 
benefit [value] of the metrics and is this value gained from the metrics worth the resources 
expended in managing, monitoring, and collecting them? The answer to this question was not 
completely discernible from any of the interviews conducted with the HCA representatives 
because for the most part they were asking of what value are the metrics to DASN ACQ. 

The majority of the metrics are collected on an annual basis. Of the seven metrics that 
the HCAs report on, only one is reported quarterly, Workforce Stability. Reporting on the 
specific metrics annually makes sense because of the nature of procurements due to yearly 
appropriations from Congress and the fiscal year cycle. However, there are two mind-sets at 
odds with each other. The first is that sometimes employees are not in their positions long 
enough for annual metrics to be of any value in a managerial sense because managers might only 
see one or two sets of annual data. Seeing limited data does not provide managers with much 
value and does not give them the ability to make decisions and manage based on the variances 
observed. The counter to this argument is that if there was past data, then that could be looked at 
in conjunction with current data. The second mind-set is, as stated above, collecting the data is 
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not a minor task and answering data calls more than once a year for specific metrics would be 
extremely time-consuming if not impossible without additional resources in place. 

The PUC metrics ( Large Contracts & Simplified Acquisition Procedures ) were discussed 
more frequently during the interview process than other metrics. The general concept of the 
metric was understood, but the HCA representatives did not see much value in the output. An 
interviewee stated, “Concept of PUC is nice, but real world application is questionable. Goes 
back to how it is used. [A result of] .003 down, why did that happen?” The HCAs need more 
feedback on the two PUC metrics and what the results mean. The use of the Customer 
Satisfaction Index was a point of contention with one SYSCOM. There were two parts to this 
point. One, this metric is already used independently of the PUC metrics. Second, the weighting 
of the Customer Satisfaction Index is increased by squaring it which increases the “role” of this 
subjective and incomplete metric. 

Another issue regarding these two metrics ( PUC-Large Contracts & PUC-Simplified 
Acquisition Procedures) involved the use of contract action dollars in the formula. Two of the 
HCA representatives from the SYSCOMs would prefer to use “contract actions” instead of 
“contract action dollars” as one of the variables in the metric. One other SYSCOM was strongly 
against “contract actions” replacing “contract action dollars.” This SYSCOM felt this change 
would not create a more accurate reflection of their performance in this area, stating, “Clearly do 
not use actions.” Another SYSCOM felt that there was not a clear answer, stating, “When you 
use dollars you could be seen as more efficient without being. [There is] not much control over 
number of actions because a lot of control is set by Congress’ appropriations.” 

Indication of the metric’s link to value was the SYSCOMs view that the Navy 
Procurement metrics are not the specific ones that each individual SYSCOM would have come 
up with to tell the story of the health of their specific organization, but that was not the purpose 
of these metrics. One SYSCOM stated, “We wouldn’t have come up with all of these metrics if 
you asked, ‘how [our SYSCOM] is doing?”’ 

D. ADDITIONAL ANALYSIS 

It became clear to the author through the evaluation of the metrics that there were two 
different types of analysis, design and use. The evaluation of the metrics’ alignment with 
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strategy, effective measurement (objectivity and completeness), and link to value, all fall under 
the “design” of this management control system. The analysis of design captures the 
fundamental structure of the system and answers the questions: Is it valid? Is the design 
sufficient enough for the system to work? This section discusses findings related to the “use” of 
the metrics. This includes, but is not limited to, issues with communication and information 
gathering. 

1. Customer 


Metric 


Customer Satisfaction Index 


Since the individual SYSCOMs are responsible for the customer survey that eventually is 
used as input into the Customer Satisfaction Index , it is important that they understand all of the 
factors that affect the survey’s validity. Not all of these factors can be controlled. There are two 
issues which are more relevant to as well as easier controlled by the SYSCOMs. The issues are: 
1) ensuring that the survey goes out to the customers every year and 2) ensuring that the DASN 
ACQ general customer satisfaction themes are covered. These two issues, out of all the ones 
mentioned, are the most important in receiving relevant data for the metric. 

2. People 


Metrics 

Employee Survey 


Workforce Stability 


Continuous Learning 


DAWIA Certification 


Acquisition Professional Community 


What was striking about the Employee Survey is what was learned through the interviews 
with the SYSCOM points of contact for the DoN Procurement Metrics. The response rate for the 
Employee Survey is less than the targeted. One SYSCOM gave an estimated response rate of 
about 9 percent and another of about 2.5 percent. It is striking to find that out of four-hundred 
surveys sent out, ten were completed. It does not matter how well the survey is written, 
employee satisfaction cannot be effectively measured with such low response rates. 


An issue with the Workforce Stability metric, as well as with others, is the availability of 
the information. It would seem simple enough to find out how many employees are onboard and 
how many are authorized, but this is not the case for every SYSCOM. A few of the SYSCOMs 
indicated that this seemingly undemanding task is extremely time consuming for them. This 


34 





goes back to the issue of expending effort and resources to what cost and what benefit. This also 
affects the objectiveness of this metric. Is the data accurate? One interviewee indicated this 
about the military portion of the Workforce Stability metric: “[We] just do not have the military 
data, we could estimate it, but then it becomes subjective.” 


3. Process 


Metrics 

E2E Metric 


P-Card Delinquency 


DD1716 


Interest Penalties 


Cycle Time 


Consolidate Service Contract 


When measuring the number of days turnaround time ( DD1716 metric) from receipt to 
disposition there may be some room for gaming the system with respect to when exactly the 
Contract Deficiency Report was received and acted upon. The different players in the process 
may have different views on when that occurred and the individuals who control the input into 
the NAFI website will win that argument and could possibly be inputting data that is not 100 
percent accurate. 

The same sort of “gaming” problem exists with the Cycle Time metric. It again, is a 
“days” measure and it relies on four definitions of when the days should start to be counted and 
two definitions of when the days should stop being counted. The Cycle Time start date is defined 
as when the earliest of the four events listed in the Metrics Guide occurs (date of Procurement 
Planning Conference (PPC), date of planning meeting equivalent to a PPC, date specialist begins 
work on procurement, or Procurement Request (PR)/Procurement Initiation Document (PID) 
receipt date) and the end date is listed as either the contract award date or for work in process, 
the current date if more than 365 days have elapsed since the start date. This may seem 
relatively straightforward, but there appear to be ways to manipulate the Cycle Time metric. For 
instance, the actual start date can be manipulated by the individual recording it to start later in 
order to appear to have a shorter cycle time. 


35 




4. Financial 


Metrics 

Procurement Direct/Indirect Ratio 


Industry Spend Analysis—Competition 


Industry Spend Analysis—Small Business 


Industry Spend Analysis—Commercial Items—Actions 


Industry Spend Analysis—Commercial Items—Dollars 


Industry Spend Analysis—Performance Based Services—Actions 


Industry Spend Analysis—Performance Based Services—Dollars 


What is difficult with regards to the Procurement Direct/Indirect Ratio metric, as well as 
others, is that the Information Technology and Human Resources system that is in place was not 
designed to generate these metrics. In one of the interviews it was indicated that the metrics 
working group which created these metrics worked with IT and HR contacts to come up with 
“friendly” metrics that would be easier to produce. However, several of the interviewees 
indicated that getting the necessary data was difficult at best and that the information is not 
broken down into the categories that are required by the metrics. For example, the salary data 
required for the Procurement Direct/Indirect Ratio is not available already categorized as direct 
or as procurement dollars. Time is limited and the interviewees indicated the extremely 
cumbersome nature of producing the required metrics. The difficulty in collecting this metric 
may vary among the SYSCOMs depending on what resources they have available and how their 
data is kept. 

The ability to obtain the HCA total procurement salary dollars, for the Procurement 
Direct/Indirect Ratio metric, was described as a complex process. In the glossary section of the 
PMG, it states that “data available to activities vary greatly, no specific way of calculating 
procurement salary dollars is mandated. These approaches should provide a reasonable estimate 
of actual labor costs, but may be based on models and/or average rates, etc” (PMG, vl4.3, p. 20). 
This means that each HCA could calculate procurement salary dollars in a different manner. If 
the HCAs are internally consistent in the manner they collect the salary data and their method of 
estimation then trends can be analyzed. They can be analyzed because the consistency of the 
collection method makes the data points comparable. This will help take the subjectivity out of 
the total procurement salary dollars variable. 

Table 2 is a summary of the findings regarding the three questions described as the 
general framework for evaluating the metrics. There are varying levels of objectivity and 
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completeness as previously discussed for each individual metric. For purposes of providing a 
summary table, “yes” or “no” was used to describe whether or not the metric was deemed 
objective or complete. The issue of the metric’s link to value was covered only for the seven 
metrics the HCAs collect. As stated above the interviewees are of the opinion that the fourteen 
metrics collected by DASN ACQ are currently of little value to the HCAs. The remaining seven 
metrics provide some value, but it is difficult to glean the extent of this value because none of the 
interviewees indicated specifically how and when they are able to use them. 


FOCUS 

AREA 

TIER 1 METRIC 

TIER 2 METRIC 

TIER 3 
METRIC 

TIER 4 
METRIC 

LINKED TO 
STRATEGY 

OBJECTIVE 

COMPLETE 

RESPONSIVE 

LINKED TO 
VALUE 

Customer 

Customer Satisfaction Index 




YES 

NO 

NO 


YES 

People 

Human Capital Index 

Employee Survey 



YES 

NO 

NO 


Workforce Stability 



YES 

YES 

NO 

SOMEWHAT 

Qualifications 

Continous 

Learning 


YES 

YES 

YES 


DAWIA 


YES 

YES 

YES 



Acquisition 

Professional 

Community 


YES 

YES 

YES 



Process 

Process Improvement Index 

E2E Metrics 



YES 

YES 

YES 



Efficiency 

Improvement 

P-Card 

Delinquency 


YES 

YES 

YES 



DD1716 


YES 

NO 

YES 



Interest 

Penalties 


YES 

YES 

YES 


SOMEWHAT 

Cycle Time 



YES 

NO 

YES 

Consolidate Services 
Contract 



YES 

YES 

YES 



Financial 

Contribution Index 

Procurement 



YES 

NO 

YES 


SOMEWHAT 

Industry Spend 
Analysis 

Competition 


YES 

YES 

YES 

tguEIIISUMlffESM 


YES 

YES 

YES 



Commercial 

Items 

Actions 

YES 

YES 

YES 



Dollars 

YES 

YES 

YES 



Performance 
Based Services 
Contracting 

Actions 

YES 

YES 

YES 



Dollars 

YES 

YES 

YES 



Value 

Product Unit Costing (PUC) 

Large Contracts 



YES 

NO 

YES 


NO 

Simplified Acquisition 
Procedures 



YES 

NO 

YES 


NO 


Table 2. DoN Procurement Metrics Evaluation Summary 

Overall, the author’s conclusion is that the metrics are not providing the type of value that 
performance metrics should for managers in an organization. Based on the interviews, it does 
not appear that the metrics are being used to make any decisions about the organizations and how 
they operate. This is due to the fact that fourteen of the twenty-one metrics are not seen, so they 
are in essence off the table, and the other seven have varying levels of value to the HCAs, if any 
at all. The seven metrics that the HCAs report on (Table 3) are of limited value for two primary 
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reasons. First, it was indicated in the interviews that estimations of variables may be used in 
order to report the metric. This not only creates subjective measurements but also causes 
inconsistency over time decreasing the validity of analyzing trends. Second, while annual 
metrics are a plus because they lessen the amount of resources the HCAs have to expend, they 
are not of much value when decision makers rotate to a new position and maybe only see one or 
two sets of completed metrics. 


Focus Area 

Metric 

Customer 

Customer Satisfaction Index 

People 

Employee Survey 


Workforce Stability 

Process 

Cycle Time 

Financial 

Procurement Direct/Indirect Ratio 

Value 

PUC—Large Contracts 


PUC—Simplified Acquisition Procedures 


Table 3. HCA Collected Metrics 


Analysis Not Specific To One Focus Area 

Since the HCAs collect and report on seven of the DoN Procurement Metrics, this 
focuses the SYSCOMs on seven of the twenty-one metrics. This being said, it is important to 
note that not having to collect the data for the other fourteen metrics is a very good thing in the 
HCAs view. They would just like feedback on them. “All the time [we have] given information, 
we have never received feedback,” was a comment during one of the interviews. When another 
HCA representative was asked if the metrics help them in any way, the response was, “They 
could, but [we receive] no feedback.” Some even thought it valuable to see the outputs of the 
other SYSCOMs, realizing that it is difficult to compare due to the differing nature of the 
procurements. Nonetheless, a few of the interviewees were still curious. One interviewee 
expressed the concern that, “comparing is dangerous.” 

Feedback was mentioned as part of the Cybernetic Feedback Model discussed in Chapter 
2. The feedback specific to the individual metrics deals with the variance from the standard 
created or the original baseline. Besides the Cybernetic Feedback Model type of feedback, 
which offers insight into deviations from the standard, the HCAs desire, as noted above, more 
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general feedback on the status of the metrics program. The interviewees asked if the 
procurement metrics are going to be cheer leaded (have a champion) and supported by DASN 
ACQ. If DASN ACQ does support the metrics then the interviewees are also asking what 
resources will be available to the HCAs in order to better comply with and complete the data 
calls in a timely and accurate manner? One quote from the interviews addresses this point: “If 
you are going to do it [use the metrics], do it consistently and bring it to attention, that will bring 
more meaning to it.” The idea of consistency is reflected in another interview, “stay consistent 
.. .there seems to never be consistency.” 

Overall the feeling was that the metrics, while not the ones each specific SYSCOM 
would have picked to explain their individual success, could have some value if they got the 
appropriate feedback from DASN ACQ. One of the interviewees expressed the idea that DASN 
ACQ should provide feedback in the following manner, “We’re seeing a downward trend [in a 
specific area/metric], and you may want to look at this, this, and this.” 

Right now there is not the sense that DASN ACQ is there to help the SYSCOMs. The 
SYSCOMs need to see a culture of “we’re trying to help you” from DASN ACQ to the 
SYSCOMs. Along the same lines one interviewee stated, “[We] need it as a system to help the 
SYSCOMs, not the SYSCOMs fearing they would get in trouble. Show how they are benefiting 
the SYSCOMs.” 

The HCA representatives stated that the fourteen metrics collected by DASN ACQ are of 
little or no use to the HCAs because they aren’t getting any feedback on them. DASN ACQ has 
indicated that part of this reason is because the set of metrics is not a finished product and DASN 
ACQ feels reluctant to put out all of the results of their data pulls when they are not fully 
convinced that the metrics are the correct ones. 
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V. CONCLUSION 


A. SUMMARY 

The Department of the Navy Procurement Metrics show a clear link to the intended 
strategy of the five Focus Areas. Each metric links back to their respective focus area and is 
aligned with Department of the Navy as well as Department of Defense strategies and goals. As 
stated in the literature review, when a metric is created, the managers creating it must know that 
by doing so they are telling their employees what is important. The simple fact that something is 
being measured speaks volumes as to its importance. 

The motivation behind each one of the metrics is important. Each metric is a piece in the 
puzzle that when put together should have the ability to provide an accurate picture of the health 
of the Navy’s procurement function. The metrics dashboard has the potential to be a great 
management by exception tool. However, this tool is only effective when those using it have 
confidence that the metrics in place are the correct ones linking to each Focus Area, have been 
collected and reported on accurately, are responsive to manager actions and decisions, provide 
the necessary information for managers to make those decisions, and have appropriate targets set 
for each metric. The proper use of the metrics dashboard by top management is not to analyze 
every area, but to just make note of areas that are not “green” and ensure that people execute a 
plan of corrective action if they have control to do so. 

Each of the metrics has had a preliminary analysis done on it that has indicated that all of 
them align with the intended strategy. The analysis continued with an evaluation of their 
objectivity and completeness. The level of objectivity and completeness varied. More analysis 
has to be done from a procurement expert’s viewpoint to see what can be done to minimize the 
subjectivity that was found to exist in some of the metrics, specifically the customer and 
employee surveys, DD1716 , Cycle Time, and the two PUC metrics. Most of the metrics were 
found to be complete; they fully explained what it is they set out to measure, but some are not 
(i.e., Customer Satisfaction Index and Employee Survey). One of those which is not complete is 
Workforce Stability. It is clear that this metric needs some rework as mentioned in section B of 
Chapter 4. The two survey metrics were evaluated as incomplete mainly because of the 
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subjectivity issues listed in Chapter 4 and also because of the response rate. An employee survey 
that has a 2.5 percent response rate is highly unlikely to give a complete picture of employee 
satisfaction. As stated before, responsiveness was not evaluated by the author. 

The value of the metrics to procurement managers was also analyzed. Most of the 
conclusions drawn about the value of the metrics came from the interviews conducted with the 
HCA representatives from six of the ten major SYSCOMs. It is unclear as to whether other 
personnel within the HCAs use the current metrics in a way that has value to them or have ideas 
for certain metrics they would like to use. Currently, according to the interviewees, fourteen of 
the twenty-one metrics provide little value to the HCAs because in their view there is no 
feedback on the metrics that are produced by DASN ACQ. The remaining seven metrics do not 
provide information that is of great value to the HCAs either. This is likely to remain the case 
until a trend can be analyzed. 

The idea of additional metrics is not one that the HCA representatives embraced. This 
goes back to the issues of value provided and cost to provide that value. One quote from an 
interview relays this point fairly succinctly, “Any additional metrics need to be evaluated on how 
much work that it is going to require and of what value it is. Even collection of how many 
people you have compared to end strength [is cumbersome].” For a few of the HCA 
representatives, the opposite idea, getting rid of the metrics, was something they wouldn’t lose 
any sleep over. 

B. RECOMMENDATIONS 

If the DoN Procurement Metrics are to be used and be successful, they need a champion. 
Partial buy in will only prolong the period of indecision on the metrics, engender confusion as to 
the status and the purpose of the metrics, and potentially waste valuable time and resources that 
could otherwise be used more effectively. 

The Procurement Metrics themselves are generally good measures of the Navy’s 
Procurement Function. The purpose of these metrics is to roll up data from all of the major 
SYSCOMs into a usable diagnostic tool. There is no perfect set of metrics that every SYSCOM 
would have agreed on as telling a complete and accurate picture of their procurement activities. 

A recommendation in this area would be to contact the appropriate person at each HCA to find 

42 



out which of the metrics they (the HCA) deem valuable for her or his SYSCOM. It could be as 
simple as a table to be completed like the one below. The table does not list all metrics or 
SYSCOMs. It just provides an example of format. The actual table would list all twenty-one 
metrics and each of the ten SYSCOMs, who would mark the valuable metrics: 



SYSCOM 1 

SYSCOM 2 

SYSCOM 3 

SYSCOM 4 

SYSCOM 5 

P-card Delinquency 

X 

X 

X 

X 


E2E 


X 


X 


Cycle Time 



X 


X 


Feedback has to be initiated by DASN ACQ with respect to the metrics. This paper may 
offer DASN ACQ some insight into the metrics themselves, the issues that the HCAs have with 
specific metrics, as well as the metrics initiative in general. Right now the feedback loop is not 
connected, it is open. The HCAs need a sense of the status of the metrics and feedback on not 
only the fourteen metrics that are being collected by DASN ACQ, but all the metrics. 

If DASN ACQ decides to continue with the DoN Procurement Metrics initiative, then the 
HCAs desire the necessary resources to do an effective job at collecting and reporting the 
metrics. Again, the level of effort currently expended and resources already available to the 
HCAs most likely differ from SYSCOM to SYSCOM. Two interviewees from separate 
SYSCOMs specifically mentioned that larger SYSCOMs have contractors to help with surveys 
while they did not. 

A new working group needs to be established. This working group should not only be 
comprised of the HCA representatives for the metrics, but also some of the managers in the 
HCAs who would specifically be using and benefiting from the metrics. A metric by metric 
evaluation can be done in much the same way as this paper evaluated the metrics. This 
evaluation would be done by operators and managers who have been in the procurement field 
and who know the ins and outs of the processes. This means that they could take what has been 
started here and fine tune it because of their experience and expertise in the procurement field. 

Several issues should be addressed when this working group meets: 
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1. Reduce the subjectivity of the indicated metrics (Customer Satisfaction Index and 
Employee Survey ) to an acceptable point. 

2. Address the completeness of the metrics with specific emphasis on Workforce 
Stability. 

3. Ascertain the actual metric responsiveness to manager and employee decisions. 

4. Discuss the nature of the feedback that will be supplied to the HCAs. 

5. Determine what additional resources are available to the HCAs from DASN ACQ 
in order to provide the most accurate and useful information. 

6. Discuss the definitions of the variables used in the metrics, specifically those 
collected by the HCAs, in order to reduce any guesswork. 

7. Discuss how the actual data needed is stored and what has to be done to 
categorize and retrieve that data to put it in a usable form for input into the metric 
formulas. 

Addressing these points by DASN ACQ and the HCAs is likely to provide very meaningful 
information on the next step for the metrics initiative. 

The results of this paper and especially the outcome of this new working group should be 
pivotal in deciding the future of the DoN Procurement Metrics. An improved understanding of 
the metrics as well as an improved set of metrics can only be achieved if this issue is approached 
openly with candid discussion of the issues with no fear of repercussions. DASN ACQ needs to 
reassure the HCAs that they are there to help. Buy in at the top should have a positive trickle 
down effect. That coupled with a renewed sense of purpose and understanding of the metrics has 
the potential to transition these metrics into the Diagnostic Control System it intended to be. 
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